In Figure 1 we demonstrate the common neighbor (CN) distribution among positive and negative test samples for ogbl-collab, ogbl-ppa, and ogbl-citation2. These results demonstrate that a vast majority of negative samples have no CNs. Since CNs is a typically good heuristic, this makes it easy to identify most negative samples. We further present the CN distribution of Cora, Citeseer, Pubmed, and ogbl-ddi in Figure 3. The CN distribution of Cora, Citeseer, and Pubmed are consistent with our previous observations on the OGB datasets in Figure 1. We note that ogbl-ddi exhibits a different distribution with other datasets. As compared to the other datasets, most of the negative samples in ogbl-ddi have common neighbors. This is likely because ogbl-ddi is considerably denser than the other graphs.

Reliable yet efficient evaluation of generalisation performance of a proposed architecture is crucial to the success of neural architecture search (NAS). Traditional approaches face a variety of limitations: training each architecture to completion is prohibitively expensive, early stopped validation accuracy may correlate poorly with fully trained performance, and model-based estimators require large training sets. We instead propose to estimate the final test performance based on a simple measure of training speed. Our estimator is theoretically motivated by the connection between generalisation and training speed, and is also inspired by the reformulation of a PAC-Bayes bound under the Bayesian setting. Our model-free estimator is simple, efficient, and cheap to implement, and does not require hyperparameter-tuning or surrogate training before deployment. We demonstrate on various NAS search spaces that our estimator consistently outperforms other alternatives in achieving better correlation with the true test performance rankings. We further show that our estimator can be easily incorporated into both query-based and one-shot NAS methods to improve the speed or quality of the search.

We pay special attention to the setting where the sample size n is small. This type of tests carries concrete significance in scientific studies.

This problem is referred to as observational overfitting [36].

Hyperparameter optimization is crucial for obtaining peak performance of machine learning models.

Appendix: How does a Neural Network's Architecture Impact its Robustness to Noisy Labels? In this section, we include additional experimental results for the predictive power in (a) representations from randomly initialized models (Appendix A.1), (b) representations learned under different We first evaluate the predictive power of randomly initialized models (a.k.a., untrained models), and Notice that lower test MAPE denotes better test performance.Model T est MAPE Random Trained Max-sum GNN 12.74 0.57 0.37 0.08 In previous experiments (section 4.2), we have shown that the predictive power in well-aligned MAE, is more helpful in learning more predictive representations under smaller noise ratios. The predictive in the representations grows as the mutual information between the noisy labels and original clean labels increases for models well-aligned with the target function. Clean Labels (DwC) and further measure the predictive power in representations learned by DwC. Table 6: Test accuracy (%) on CIF AR-10 with flipped label noise .

Noisy labels are inevitable in large real-world datasets.